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SORTING AND PREIMAGES OF PATTERN CLASSES 

ANDERS CLAESSON AND HENNING ULFARSSON 



Abstract. We introduce an algorithm to determine when a sorting operation, such as 
^^ ' stack-sort or bubble-sort, outputs a given pattern. The algorithm provides a new proof 

of the description of West-2-stack-sortable permutations, that is permutations that are 
completely sorted when passed twice through a stack, in terms of patterns. We also 
solve the long-standing problem of describing West-3-stack-sortable permutations. This 
requires a new type of generalized permutation pattern we call a decorated pattern. 
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1. Introduction 

'k>( ' The set of permutations of {1, . . . ,n} is denoted Sn- Permutations will be written in 

^ . one-line notation, and the identity permutation 12 ■ ■ -n will be denoted id„, or just id if 

n is understood from context. 

In the 1970's Knuth |Knu75] initiated the study of sorting and pattern avoidance in 
permutations. He considered the problem of sorting a permutation by passing it through 
a stack. A stack is a last in, first out data structure with two fundamental operations: the 
push operation moves an item from the input to the top of the stack; the pop operation 
moves an item from the top of the stack to the output. Consider trying to sort the 
permutation 231 by a stack, as shown in Figure [H 

Note that we always want the elements in the stack to be increasing, from the top, 
since otherwise it would be impossible for the output to be sorted. We failed to sort 
the permutation in one pass and therefore say that it is not stack-sortable, which Knuth 
showed is equivalent to containg 231 as a pattern. We will reprove this in Theorem 13.11 
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Figure 1. Trying, and failing, to sort 231 with a stack. The figure is read 
from right to left 



Several variations on Knuth's original problem have been considered, see Bona |B6n03] 
for a survey. In this paper we consider repeatedly passing a permutation through a stack, 
while keeping the elements on the stack in increasing order from top to bottom. We also 
consider the bubble-sort operator. We introduce a new method for finding patterns in a 
permutation that will cause these sorting devices to output a given pattern, and use this 
to solve the long-standing problem of describing West- 3- stack-sort able permutations. If 
the given pattern is classical (defined below) we show that the mesh patterns introduced 
by Branden and the first author [BCllj suffice, but if the given pattern is itself a mesh 
pattern we will need to introduce a new kind of generalized pattern we call a decorated 
pattern. 

In Section m we describe an algorithm that given a classical pattern p produces a finite 
list of (marked) mesh patterns P such that S^^{Av{p)) = Av(P), where S is the stack- 
sort operator. This algorithm automates proving some of the statements in the previous 
sections and can be extended to cover the bubble-sort operator as well. 

We hope that our work is a step towards the general comprehension of the stack-sort 
operator and other similar operators. This extend ed abstract will appear at FPSAC 2012 
and is based on two papers of the authors Ulla[ ICUllj . 



2. Generalized permutation patterns 

A standardization of a list of numbers is another list of the same length such that the 
smallest letter in the original list has been replaced with 1, the second smallest with 2, etc. 
The standardization of 5371 is 3241. A classical (permutation) pattern is a permutation 
p in Sk- A permutation vr in 5„ contains, or has an occurrence of the pattern p if there 
are indices 1 < ii < ■ ■ ■ < ik "^ n such that the standardization of vr(zi) ■ ■ ■Ti{ik) equals 
the pattern p. If a permutation does not contain a pattern p we say it avoids p. The 
permutation vr = 526413 contains the pattern p = 132, and has three occurrences of it, 
in the subsequences 264, 263 and 243. We can draw the graph of the permutation by 
graphing the coordinates (i,7r(z)) on a grid. For example the permutation tt above is 
shown in Figure [2] where we have circled the occurrences of the pattern p. 
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Figure 2. The permutation 526413 and three occurrences of the pattern 132 
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The same permutation avoids 123, since we cannot find an increasing subsequence of 
lengtli tliree. 



2.1. Mesh patterns and barred patterns. Mesh patterns were introduced in [BCllj . 

We review them via an example. The mesh pattern ^^^ occurs in a permutation if we 

can find the underlying classical pattern 132 positioned in such a way that the shaded 
regions are not occupied by other entries in the permutation. Another way of writing a 
mesh pattern is to give the underlying classical pattern, followed by the set of shaded 
boxes, labelled by their lower left corner (the left-most box in the bottom-most row being 
(0, 0)). This mesh pattern is (132, {(0, 2), (1, 2), (2, 2)}). 

Consider the permutation 526413. From above we know that the classical pattern has 
three occurrences in this permutation. In Figure [3] one can see that just one of these 
satisfies the additional requirement that there be no additional entries in the shaded 
region "between and to the left of the 3 and the 2" . 
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Figure 3. The permutation 526413 and one occurrences of the mesh pat- 
tern (132, {(0,2), (1,2), (2, 2)}) 



Barred patterns, introduced by West |Wes90] . are classical patterns with bars over some 
of the entries. Such a pattern is contained in a permutation if the standardization of the 
unbarred entries is contained in the permutation in such a way that they are not part 
of an occurrence of the whole pattern. The permutation 5264173 contains the barred 
pattern 35241, in the subsequence 5473, since that is an occurrence of 3241 that is not 
part of an occurrence of 35241. In |BCllj it was shown that any barred pattern with one 
barred entry is a mesh pattern. The barred pattern we discussed here is in fact the mesh 
pattern below. 
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2.2. Marked mesh patterns and decorated patterns. Marked mesh patterns were 
introduced by the second author in jUllbj and give finer control over whether a certain 
region in a permutation is allowed to contain elements, and if so, how many. Again we 



just give an example. Consider the marked mesh pattern 




. The meaning of the 1 in 



the region containing boxes (1, 0), (1, 1), (2, 0) and (2, 1) is that this region must contain 
at least one entry. In Figure H] we see that there is exactly one occurrence of this mesh 
pattern in the permutation 526413. 

Marked mesh patterns will be useful when we need to add elements into an existing 
pattern to ensure other elements are popped by a particular sorting device. Below we 
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Figure 4. The permutation 526413 and one occurrence of a marked mesh pattern 

will need even finer control over what is allowed inside a particular region in a pattern. 
We will need to control whether the entries in the region contain a particular pattern. 



Consider for example the decorated pattern Z*^^ • The decorated region in the middle 



signifies that an occurrence of this pattern should be an occurrence of the underlying 
classical pattern 21 that additionally does not have entries in the region that contain the 
pattern 12 - or equivalently - whatever is in that region must be in descending order, 
from left to right. In Figure |5] there is an occurrence of the decorated pattern on the left 
and on the right we have an occurrence of the classical pattern 21 that does not satisfy the 
requirements of the decorated region. Below we state the formal definition of a decorated 
pattern. 



id± 
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t 
Figure 5. The permutation 526413 and one occurrences of a decorated pattern 

For integers a < 6 we use |a, 6] to denote the set {a, a + 1, . . . , 6}. 

Definition 2.1. A decorated •pattern {p,C) of length k consists of a classical pattern p of 
length k and a collection C which contains pairs (C, q) where C is a subset of the square 
|0, kj X |0, kj and q is some pattern, possibly another decorated pattern. An occurrence of 
{p, C) in a permutation vr is a subset u of the diagram G('7r) = {{i, 7r(i)) \1 < i < n} such 
that there are order-preserving injections a, (3 : |l,fc] — )■ |l,n] satisfying two conditions: 

{l)u = {{a{t),P{j)):{t,j)eG{p)}. 

(2) Let R,j = la{i) + 1, a{i + 1) - 1] x |/3(j) + 1, f3{j + 1) - 1], with a(0) = = /3(0) 

and a{k + 1) = n + 1 = f3{k + 1). For each pair (C, q) we let C = [Jr^ -s^^ Rij and 

require that C fl G^tt) avoidsg. 



3. Finding preimages of patterns 

In this section we define a method for describing patterns that are guaranteed to produce 
a given pattern in a permutation after it is sorted by a stack or with the bubble-sort 
operator. 
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3.1. The stack-sort operator. For a permutation vr we will denote with S'(7r) the image 
of vr after it is passed once through a stack. Note that a permutation vr in 5„ is stack- 
sortable if and only if S'(vr) G Av„(21), where Av„(21) is the set of permutations of length 
n that avoid 21. Of course Av„(21) = {id} but framing the definition like this leads 
to a generalization: Given a pattern p, what conditions need to be put on vr such that 
S{'k) G Av„(p). Exploring a sorting operator from this angle was first done by Albert et 
al. |AAB"'"11] : they, however, studied the bubble-sort operator rather than the stack-sort 
operator. 

Below we will call permutations such that 5''^(vr) = id, West-k- stack- sortable permuta- 
tions, since West considered this generalization from the case of one stack first. Note that 
for k > 1 these permutations are different from the k- stack- sortable permutations, which 
are the permutations that can be sorted by using k stacks in series without the require- 
ment that the elements on the stack are increasing from top to bottom. For example, the 
permutation 2341 is not West-2-stack-sortable, but if we put the entries 2,3,4 onto the 
first stack, pass 1 all the way to the end, and then use the second stack to sort 2, 3, 4 we 
end up with 1234. So 2341 is 2-stack-sortable. 

The basic idea behind the method we are about to describe is that S'(vr) has an oc- 
currence of a pattern p of length k if and only if the k elements in this occurrence were 
present in vr as some kind of pattern before we sorted. We start by showing how this 
idea allows us to describe stack-sort able permutations as well as West-2-stack-sortable 
permutations. 

We know that vr is not sorted by the stack if and only if S'(vr) contains the classical 
pattern 21. Therefore consider a particular occurrence of this pattern in S'(vr). Before 
sorting, the elements in this occurrence must have formed the pattern 21 = i^ in vr. In 
order to remain in this order the element corresponding to 2 must be popped off the stack 
by a larger element before the element corresponding to 1 enters. Thus the box (1, 2) 
must have at least one element and we have an occurrence of the marked mesh pattern 

[ 1 which is equivalent to the pattern rii • We have therefore reproven Knuth's result. 



Theorem 3.1 ( |Knu75] ) . A permutation is stack- sortable if and only if it avoids 231. 

We can similarly reprove West's result on West-2-stack-sortable permutations, i.e., 
permutations vr such that 5'^(vr) = id. By Knuth's result we know that vr will be sorted 
by two passes through the stack if and only if S'(vr) avoids the pattern 231. An occurrence 
of 231 must have formed either of the patterns 



in vr. For the elements in the pattern on the left to stay in the same order as they 
pass through the stack we must have an element in the box (2, 3) to pop the element 
corresponding to 3 out of the stack before the smallest element enters. Now, the opposite 
happens for the pattern on the right. The 3 must stay on the stack until 2 enters, so there 
can be no elements in the box (1,3). Then both 2 and 3 must leave the stack before 1 
enters. Thus the patterns above become the two marked mesh patterns on the left below. 
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These are more naturally written as the two mesh patterns on the right. 
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By a lemma of Hilmarsson, et al. [HJS^TI], the last pattern on the right is equivalent to 



in the sense that a permutation either contains both patterns or avoids both. (For 
these two patterns it is also easy to see directly that they are equivalent.) As mentioned 
above this pattern is another representation of the barred pattern 35241. Thus we have 
re- derived West's result. 



Theorem 3.2 ( |Wes90] ) . A permutation is West-2- stack- sortahle if and only if it avoids 
2341 and?,l2Al. 

3.2. The bubble-sort operator. The bubble-sort operator swaps adjacent entries in a 
permutation if the left entry is larger than the right entry. Let -B(vr) denote the output 
of one pass of bubble-sort on vr. For instance, 5(521634) = 215346. A modification of 
the method above works equally well for B. We see that -B(7r) contains the pattern 21 if 
and only if vr contains 21 = i^ . To make sure that these elements stay in this order, 
we either need a large element in front of 2, which would mean 2 would never be moved; 
or a large element in between 2 and 1 that will stop 2 from moving past 1. We get the 

marked mesh pattern 



which is equivalent to the two classical patterns 231 and 

321. Thus, B['k) = id if and only if vr avoids 231 and 321, as was first made explicit by 
Albert, et al. |AAB"'"11] . In the same paper the authors show that for any classical pattern 
p with at least three left-to-right maxima, the third of which is not the final symbol of 
p, the set B~^{Av{p)) is not described by classical patterns. We consider the s mallest 
example of such a pattern, p = 1243. For a proof of the following proposition see Ulla 
Proposition 3.3]. 



Proposition 3.3. 5"'(Av(1243)) = Av 
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Note that all the patterns can be expanded to mesh patterns, but at the cost of having 
more patterns. 



3.3. West-3-stack-sortable permutations. 

mutations, i.e., permutations tx such that S^[ 

SHn) 



We now turn to West- 3- stack-sort able per- 
id. By West's result (Theorem 13. 2p 



n 



id if and only if 5'(7r) avoids these two patterns 
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(3.1) 



We will use the same method as we did above, but when we consider the pattern on the 
right, the shaded box will cause some complications and the decorated patterns introduced 
above will be necessary. 
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Lemma 3.4. An occurrence of Wi in S{7t) comes from exactly one of the patterns below 
in vr. 
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Proof. Use Algorithm [T] from Section |H See also Ulla[ Lemma 4.1]. 
We now consider the pattern W2, but without the shading. 



D 



Lemma 3.5. An occurrence o/3241 in S{tt) comes from exactly one of the patterns below 
in 71. 
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Proof. Use Algorithm [T] from Section HI See also Ulla[ Lemma 4.2]. 



D 



We now consider what additional conditions cause the patterns ji, J2 and js in the 
lemma to give the correct shading in the pattern W2. We express these conditions in the 
next lemma and two propositions. 

Lemma 3.6. An occurrence of j^ in a permutation vr will become an occurrence ofW2 in 

SiTT). 

We leave the proof to the reader. We rename the pattern J3 and note that it can also 
be expanded into a mesh pattern. 



J3 



1^ 



Proposition 3.7. An occurrence of J2 in a permutation it becomes an occurrence ofW2 
in S{tt) if and only if it is part of one of the patterns below, where elements that have been 
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added to J2 are circled. 
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Proof. To ensure that there are no elements in the shaded box in W2 we must look at the 
element that pops 3 in J2- There are four different possibilities. 



J2,l 




J2,2 




J2,3 
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We explain the shadings and the decoration of the pattern j2,2 as the others are similar. 
For this pattern, the size of the element that popped 3 from the stack was in-between 
4 and 5. Since this was the element that popped 3 there can be no elements in boxes 
(1, 3), . . . , (1, 6). The boxes (2, 5) and (2, 6) cannot contain an element, since that would 
pop the element we just added (the 5 in j2,2) and this element would land in the shaded 
box in W2. Now consider the decorated box (2,4). It can contain elements, but none of 
them are allowed to leave the stack prior to 4 being pushed on, since any one of them 
would then land in the shaded box in W2. Elements in this region must then be in 
descending order. 

We must make sure that elements that arrived on the stack prior to 3 are not popped 
into the shaded box in W2. We only consider the pattern J2,2 here. If there are elements 
from box (0, 4) still on the stack when 3 is put on they will be popped by 5 and will 
land in the shaded box in W2. We must therefore have this box empty, or an element in 
box (0, 5) or (0, 6) that pops everything before 3 is pushed on. We get the three patterns 

<^2,7) <^2,8) <^2,9- n 

We now consider what constraints must be imposed on the pattern ji to get W2 after 
sorting. 

Proposition 3.8. An occurrence of ji in a permutation it becomes an occurrence ofW2 
in S{7i) if and only if it is part of one of the patterns below, where elements that have been 



SORTING AND PREIMAGES OF PATTERN CLASSES 

added to ji are circled. 
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Proof. The proof of this proposition is similar to the proof of Proposition 13 . 71 and therefore 
omitted. D 



Taken together, Lemmas 13.41 and 13. 6[ with Propositions 13.71 and 13.81 produce a hst of 
29 patterns describing permutations that are not West-3-stack-sortable. We can simphfy 
this hst considerably by observing that the patterns Ji^i, . . . , Ji^ all imply containment 
of /i so we can remove them. Further simplifications of this sort can be made. Also, 
by considering what happens with the decorated patterns when they contain a certain 
number of elements in the decorated region the list can be simplified even further. See 



Ulla[ Theorem 4.6] for the details. 



Theorem 3.9. A permutation is West-3- stack- sortable if and only if it avoids the deco- 
rated patterns 
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Note that each of the decorated patterns in the theorem is equivalent to an infinite family 
of mesh patterns. 

4. An ALGORITHM 



We shall now automate proving statements such as Theorem l3.2[ More precisely we shall 
provide an algorithm that, given a classical pattern p, produces a finite list of (marked) 
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mesh patterns P such that 

S-\kY{p)) = kY{P). 

The algorithm can be modified for the bubble-sort operator to prove statements such as 
Proposition 13.31 Given the classical pattern p we identify what orderings of the letters in 
p are possible prior to sorting, and produce a list, denoted cand(p), of candidates which 
themselves are classical patterns. 

Proposition 4.1. Let p he a permutation of a finite set of integers, and let the largest 
letter of p be m = max(p). Write p = am/S and a = aia2 ■ ■ -ai. Then 

i 

cand(j9) = M { I'fTT'S '■ 7 ^ cand(aia2 • ■ ■ ctj), S e cand(aj+i • • • ai(3) } 

j=0 

contains all classical patterns that can become p after one pass of stack-sort. 

This proposition gives a recursive algorithm for computing cand(p). Recall that an 
inversion in a permutation is an occurrence of the classical pattern 21, while a non- 
inversion is an occurrence of 12. Note that if two elements in p are part of an inversion, 
they must also be part of an inversion in all patterns in cand(p). Non-inversion in p place 
no restrictions on the patterns in cand(p). 

Proof. The idea behind the proof is that letters can only be moved to the left and small 
letters are stopped by larger letters. This implies that after the largest letter, m, has been 
moved to a particular location we can recurse on what is remaining to the left and the 
right. n 

Note that cand(132) = {321,312,132}. However, it is easy to check that there is no 
way that 321 can become 132 after sorting. Candidates like this one are removed in lines 
9-11 in Algorithm [T] below. 

Example 4.2. For the pattern p = 3241, considered in Lemma 13.51 we have m = 4, 
a = 32, f3 = 1 and i = 2. When j = 0, 7 is the empty word and we get the set 
{4(5: 5 E cand(321)}. It is easy to verify that cand(321) = {321}, so we get the pattern 
4321. When j = 1 we get the set {745: 7 G cand(3), 5 G cand(21)}. Again it is easy to 
check that cand(21) = {21} so we get the pattern 3421. Finally, when i = 2 we get the set 
{74(5: 7 G cand(32), 6 G cand(l)} which gives us the pattern 3241. In total we have the 
patterns 4321,3421, and 3241, which are the underlying classical patterns in Lemma [ 



Before we state the algorithm note that 

(1) if M > 1; is an inversion in p' that becomes a non-inversion in p = S{p'), then u 
must stay on the stack until v arrives, and therefore we must shade all the boxes 
above u and between u and v. Thus there can be no elements of p' in this shaded 
region. This is handled by line 3 in Algorithm [U 

(2) ii u > V is an inversion in p' that becomes an inversion in p = S{p'), then there 
must be another element c > u that pops u before v is pushed onto the stack, 
thus maintaining the inversion. If such an element is present in p' we need not 
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do anything. If there is no such element we need to mark the region above u and 
between u and v with a "1". This is handled by lines 4-16 in Algorithm [1] 

Algorithm 1 ShadeAndMark 
Input: A G cand(p) 



n := \p\ 

marks := 

shades := U { I-^ "^('")) -^ "^('") ~ ^l >< I"^; '^l '■ (""^ '^) ^ ninv(p) } 



4: for {u,v) G inv(p) do 

5: i := A"^(m) 

6: J := X-\v) 

7: if A(£) <Mfor alHG |i + l,jl then 

8: M := li,j - 11 X lii,nj \ shades 

9: if M = then 

10: return This candidate cannot be marked properly 

11: end if 

12: if m 2 M for all m G marks then 

13: add M to marks and remove all supersets of M 

14: end if 

15: end if 

16: end for 

17: return (A, shades, marks) 

Here ninv(p) is the set of non-inversions in p, and inv(p) is the set of inversions in p. 

Example 4.3. For the pattern p = 3241 in Lemma [3.51 we have ninv(p) = {(3,4), (2,4)} 
and inv(p) = {(3, 2), (3, 1), (2, 1), (4, 1)}. We saw in Example |0] that there are three 
candidates. We only consider A = 4321, for which A^^ = A. The shading is the union of 
the sets {1} x {4} and |1,2] x {4} which is {(1,4), (2,4)}. The for-loop in the algorithm 
proceeds as follows. 

• {u,v) = (3,2): M = {(2,3)} is added to marks, 

• {u,v) = (3,1): M = {(2, 3), (3, 3), (3, 4)} is a superset of {(2,3)}, so it is not 
added to marks, 

• {u, v) = (2, 1): M = {(3, 2), (3, 3), (3, 4)} is added to marks, 

• {u, v) = (4, 1): M = {(3, 4)} is added to marks and the superset {(3, 2), (3, 3), (3, 4)} 
is removed. 

This leaves us with the marking {{(2,3)}, {(3,4)}} which is consistent with Lemma [3.51 
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The full algorithm calls ShadeAndMark for all classical patterns in cand(p). Below is 
the result of applying this algorithm to all classical patterns of length 3. 

S-\Av{123)) = Av 




S-\Ay{132)) = Av 



S-\Ay{213)) = Av 
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5"'(Av(231)) = See Theorem O 
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Bouvel and Guibert |BG11] have found a bijection between S ^(Av(312)) and the set 
of Baxter permutations; they have also found a bijection between S'^^(Av(231)) and 
S-\Av{132)). 

We note that the algorithm can easily be extended to accept a finite list of classical 
patterns. We also note that a slight modification of algorithm ShadeAndMark has been 
shown to work with the bubble-sort operator |CU11] : this algorithm can for example be 
used to prove Proposition | 



5. Open problems 

The algorithm above describes the preimage of any set Av{p) where p is a classical 
pattern. Can the algorithm be extended to cover the case where p is a mesh pattern, 
or even a decorated pattern? Solving this problem would automate the description of 
West-3-stack-sortable permutations. More generally, is there a pattern definition that is 
stable under S*"^? 



West |Wes90j conjectured, and Zeilberger |Zei92] proved, that the number of West- 
2-stack-sortable permutations is given by 2{3n)\ / {{n + l)!(2n + 1)!). Later Dulucq, Gire 
and West |DGW96J found these permutations to be in bijection with rooted non-separable 
planar maps. The enumeration of West-3-stack-sortable permutations is completely open, 
but knowing the patterns in Theorem 13.91 could provide some insight. 
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